Device, server, system and method to generate mutual photometric effects

ABSTRACT

The invention concerns a device for generating mutual photometric effects. The invention concerns also a server for delivering photometric parameters for generating mutual photometric effects and a system including such a device and such a server.  
     According to the invention the device ( 1 ) comprises  
     means for receiving and demultiplexing said visual data sets and photometric parameters respectively associated with said data sets,  
     means for defining said mutual photometric effects to be generated for these photometric parameters,  
     composition and rendering means for positioning said visual data sets in said common support space and applying said effects defined for said photometric parameters from at least one of said visual data sets to at least one other of said visual data sets so that at least one visual data set influences one other visual data set in said common support space.

[0001] The invention concerns a method and device to generate mutualphotometric effects in a common support space between a plurality ofvisual data sets.

[0002] Applying special effects methods has been for a long time aworking field in the industry. More and more, applications, such asinteractive television, or any multimedia applications bring the need ofefficient data manipulation.

[0003] The invention concerns particularly the merging of different datasets for displaying and the mutual interaction of these different datasets. Namely, in order to improve the interaction between visual datasets, audiovisual data sets or other multi-modal data sets, it isimportant that some parameters of the different data sets modify theother data sets and among these parameters, the illumination and thereflection.

[0004] To result in a realistic or nice looking final image, mutualphotometric effects between different visual data sets have to beconsidered. This is especially true when importing a three-dimensionalobject in front of a video sequence for instance.

[0005] Various methods exist to apply mutual photometric effects tovisual data, like shading, specular reflections, cast shadows, mutualillumination.

[0006] In film production, professional tools are used to generatephotometric special effects off-line. These tools need as input thephotometric properties of the visual data as for example light sourcesor surface reflection properties. Before generating photometric specialeffects, film production experts recover the photometric propertiesvisually or the photometric properties are measured using measuringtools during film production in the real scene.

[0007] In research laboratories, photometric characteristics can berecovered today off-line by automatic algorithms of very highcomputational cost or by semi-automatic manual interaction.

[0008] The off-line or interactive recovering of photometric propertiesdoes not allow a real time treatment of the visual data sets forgenerating photometric special effects.

[0009] In the MPEG-7 (MPEG standing for “Motion Picture Expert Group”)proposal n° 99 entitled “An illumination effect descriptor for videosequences”, Stauder proposes to describe illumination propertiesimplicitly by image signal changes for content-based visual informationretrieval.

[0010] The invention allows a real time treatment of the visual datasets for generating photometric special effects, describing illuminationproperties in term of light sources for special effect generation.

[0011] In a first aspect, the invention proposes a device to generatemutual photometric effects between a plurality of visual data sets,comprising means to position the visual data sets in a common supportspace characterized in that it comprises:

[0012] means for receiving and demultiplexing said visual data sets andphotometric parameters respectively associated with said data sets,

[0013] means for defining said mutual photometric effects to begenerated for these photometric parameters,

[0014] composition and rendering means for positioning said visual datasets in said common support space and applying said effects defined forsaid photometric parameters from at least one of said visual data setsto at least one other of said visual data sets so that at least onevisual data set influences one other visual data set in said commonsupport space.

[0015] The invention allows the generation of mutual photometric effectsby using the photometric parameters that are transmitted with the visualdata sets to be displayed and so can avoid the use of complex devices ormethods needed to extract the photometric properties from the visualdata sets on the device. In fact the devices that could typicallygenerate mutual photometric effects are low cost devices or consumerproducts which price has to be reduced. So, it is a great advantage todissociate this functionality from the device and to provide means totreat the photometric parameters received from an outside terminal.

[0016] The means for defining the effects to be generated for thephotometric parameters have the capability to define how meta dataassociated to multimedia data influence other multimedia data. Itgenerates some information called “rendering modi” which represents thisinfluence. This enables the composition and rendering means to displaythe visual data sets, while applying the photometric effects on thesedata sets.

[0017] The invention is thus particularly interesting in systemsproviding meta data such as systems providing MEPG-4 coded data andMPEG-7 coded data.

[0018] According to a second aspect, the invention proposes a server(Aj) for delivering multimedia data including visual data setsassociated with respective photometric parameters for said visual datasets, characterized in that it comprises

[0019] means for managing the storage and conditioning of photometricparameters for said visual data sets,

[0020] means for transmitting over a transmission link (Cj) saidmultimedia data bundled with their associated photometric parameters,said multimedia data including means to display at least one of thevisual data sets with another of said visual data sets in a commonsupport space.

[0021] This server enables the preparation of photometric parameters tobe used for instance with a device according to the first aspect of theinvention.

[0022] However, this server can be used with any other terminal thatwould need photometric parameters as input parameters.

[0023] This server has the great interest to deliver the multimedia dataassociated with photometric parameters related to them. This avoids, ona corresponding receiver, the generation of photometric parameters.

[0024] According to a third aspect, the invention proposes a multimediasystem characterized in that it comprises

[0025] at least one server (Aj) according to any of claim 6 to 7 fordelivering multimedia data including at least two visual data setsassociated with photometric parameters for visual data sets,

[0026] means (Cj) for transmitting said visual data sets and theirrespective photometric parameters bundled together from said server toat least one device according to any of claims 1 to 5,

[0027] a device according to any of claims 1 to 5 to generate mutualphotometric effects for applying the effects of said deliveredphotometric parameters of at least one of said delivered visual datasets to at least another of said delivered visual data sets so that saidvisual data sets influences said other visual data set in a commonsupport space.

[0028] According to a fourth aspect, the invention concerns also amethod to generate mutual photometric effects between a plurality ofvisual data sets, wherein one positions the visual data sets in a commonsupport space, characterized in that:

[0029] one delivers multimedia data including at least two visual datasets respectively associated with photometric parameters for visual datasets,

[0030] one transmits, bundled together, said visual data sets and theirrespective photometric parameters to at least one managing means,

[0031] one defines dynamically said mutual photometric effects for thephotometric parameters and their associated visual data sets,

[0032] and one applies said effects defined by said photometricparameters from at least one of said visual data sets to at leastanother of said visual data sets so that said visual data set influencessaid other visual data set.

[0033] The invention concerns also a television receiver, a set-top box,or any fixed or mobile terminal having the characteristics of the deviceto generate mutual photometric effects mentioned above, the advantagesof the television receiver, the set-top box, the fixed or mobileterminal being the same as the ones of the device to generate mutualphotometric effects.

[0034] Other characteristics and advantages of the invention will appearthrough the description of a non-limiting embodiment of the invention,which will be illustrated, with the help of the enclosed drawings amongwhich

[0035]FIG. 1 represents a system according to the invention,

[0036]FIG. 2 represents a server according to the invention,

[0037]FIG. 3 represents a device according to the invention.

[0038]FIG. 1 illustrates a system for generating mutual photometriceffects according to the invention including a device 1 for generatingmutual photometric effects according to the invention connected to adisplay 6 through a connection link 7. Said device 1 is also connectedto n servers (A1, A2, . . . , Aj, . . . , An) through transmission links(C1, C2, . . . , Cj, . . . , Cn).

[0039] The servers A1, A2, . . . , Aj, . . . , An are represented onFIG. 2 and the device 1 according to the invention is represented onFIG. 3.

[0040] Transmission links C1, C2, . . . , Cj, . . . , Cn can be all ofthe same type or can be of different types. In a preferred embodiment,as represented on FIG. 1, transmission link C1 is different fromtransmission links C2, C3, . . . , Cj, . . . , Cn. In this embodiment,transmission link C1 is a broadcast link. The transmission links C2, C3,. . . , Cj, . . . , Cn are for example Internet links. This correspondsto the case when a user is watching television programs received from abroadcaster from server Al and wants to display on display 6 somepictures selected from an Internet web site he receives from one of theservers A2, A3, . . . , Aj, . . . , An.

[0041]FIG. 2 shows a bloc diagram of any of the servers A1, A2, . . . ,Aj, . . . , An. Such a server includes a database 17, a multimedia coder9, a meta data coder 10, a meta data management module 18 and atransmission module 8.

[0042] The database 17 contains multimedia data and meta data.Multimedia data can be audiovisual data streams, 3D scenes, hyperdocuments, or any kind of multi or single modal data.

[0043] The meta data represent the photometric properties of the storedmultimedia data. It can be a single light source or several lightsources of different shapes (point light, area light) and different kindof intensities (monochrome, coloured, spatially constant or spatiallyvarying). Instead of light sources, also illumination maps or globalillumination descriptors can be used. Photometric meta data can containtransparency and refraction.

[0044] Other meta data than illumination can be foreseen. For example,meta data can contain a sequence of reflectance maps. The sequence ofreflectance maps associates one reflectance map to each video image.Each pixel of the reflectance map contains one or more parametersdescribing the reflection properties of the corresponding video pixel.By such a sequence of reflectance maps, special effects like specularreflections in a video can be generated. Other representations ofreflection can be used, for example symbolic or numerical reflectionfunctions or reflectance maps combined with depth maps.

[0045] The meta data management module 18 generates and manages the metadata and associates them to the multimedia data. The meta datageneration can be offline or online. It can be user assisted orautomatic. Pre-calculated meta data can be used, the multimedia data canbe used, or other data can be used.

[0046] The meta data are associated to the multimedia data. For example,a light source is associated to a video image or to a video sequence, areflectance map is associated to a 3D object, or a reflectance map and adepth map are associated to a video image. Therefore, the meta datacontain also means to display at least another visual data set with thatvisual data set in a common support space. These means include means toestablish a relationship between meta data and multimedia data.

[0047] That relationship can be distance or relative position betweenmeta data elements and multimedia data elements. The geometricrelationship can be also represented by a geometric scene description.The geometric relationship defines how multimedia data elements such asvideo images or 3D objects and meta data such as light sources orreflection maps are positioned in a common support space. The commonsupport space can be a plane, a 3D surface, a 3D space or any otherspace.

[0048] That relationship also describes, whether a meta data element hasthe capability to influence a multimedia data element or not. Forexample, a light source representing the illumination of a video imagecan influence another multimedia data element but not the video imageitself.

[0049] That relationship can refer to multimedia data elements and metadata elements of the data base 17 or to generic data elements that arenot known to the server Aj. By this, multimedia data elements fromserver Aj and multimedia data elements from other servers can bedisplayed on the display 6 of device 1 in a common support spaceinfluencing themselves mutually by consideration of their photometricproperties such as illumination and reflection contained in the metadata.

[0050] The management module 18 manages the meta data. For example, itcan decide to transmit or not a certain meta data associated to amultimedia data that is demanded by device 1. It defines also the typeof coding mode, the kind of association of the photometric parameters tomultimedia data. These decisions may depend on the type and state of thetransmission link Cj, on the type and state of device 1, on the type orquality level of service, on pricing and payment mode, or on othercriteria such as the profile of user using device 1. Module 18 may alsodecide on spatial resolution, numerical precision or size of meta datato be generated or to be transmitted. Management module 18 can do othermanagement tasks related to the generation of meta data, storage of metadata in the database, coding of meta data or transmission of meta data.

[0051] The type of the multimedia coder 9 depends on the kind ofmultimedia data and may be a video coder according to the MPEG-4 simpleprofile, a 3D data coder as MPEG-4 BIFS (standing for “BinaryInformation for Scene”) or any other kind of multimedia data coder.

[0052] The meta data coder 10 compresses the meta data and depends onthe kind of meta data. In case of reflection maps, the meta data codercan contain a video coder, for example MPEG-4 simple profile, MPEG-4core profile or JPEG (standing for “Joint Picture Expert Group”). Forillumination parameters, the meta data decoder can contain an MPEG-4BIFS module or a VRML (standing for “Virtual Reality Modeling Language”)module. The meta data decoder can contain other modules as floatingpoint number coder, ASCII (standing for “American National Standard Codefor Information Interchange”) coder, or any other kind of coder.

[0053] The transmission module 8 depends on the kind of transmissionlinks C1, C2, . . . , Cj, . . . , Cn used and on the type of device 1used. The transmission module 8 performs transmission and receptionrelated tasks as modulation, demodulation, multiplexing, synchronizationand error correction. Other transmission related operations arepossible. Transmission links C1, C2, . . . , Cj, . . . , Cn transmitmultimedia data, photometric meta data and signalling information.

[0054] On FIG. 3, a device 1 for generating mutual photometric effectsaccording to the invention is represented. Said device includes abrowser 2 that is connected to a memory 3, to a composition/renderingmeans 4, to an interface 5, to a multimedia decoder 12, a meta datadecoder 13 and a meta data management module 16 through a communicationbus 15.

[0055] The browser 2 is an interface with a user, which allows the userto select the multimedia data he wants to display on display 6. Theselected multimedia data are requested via the interface 5 and therequest is sent through communication links C1, C2, . . . , Cj, . . . ,Cn, respectively, to servers A1, A2, . . . , Aj, . . . , An. The browser2 can be accessed through a remote controller, through a keyboard or anyman machine interface (not represented on the drawings). The browser canbe for example a graphical interface.

[0056] The memory 3 stores multimedia data, meta data and browserrelated parameters received from the servers A1, A2, . . . , Aj, . . . ,An and to be displayed on display 6.

[0057] The bus 15 is used for interconnecting the different modules ofthe device 1.

[0058] A module 11 includes said multimedia decoding module 12, saidmeta data decoding module 13, said interface 5 and a transmission module14.

[0059] Multimedia decoder 12 decodes the multimedia data coded by themultimedia data coder 9 of one of the servers A1, . . . , Aj, . . . ,An.

[0060] Meta data decoder 13 decodes the meta data coded by the meta datacoder 10 of one of the servers A1, . . . , Aj, . . . , An.

[0061] The transmission module 14 depends on the kind of transmissionlinks C1, C2, . . . , Cj, . . . , Cn and kind of servers A1, . . . , Aj,. . . , An used. The transmission module 14 performs transmission andreception related tasks as modulation, demodulation, multiplexing,demultiplexing, synchronization and error correction. Other transmissionrelated operations are possible.

[0062] The meta data management module 16 manages the meta data receivedby the communication links C1, C2, . . . , Cj, . . . , Cn and stored inthe memory 3. Management tasks are for example the spatial resolutionand numerical precision of the stored meta data, the decision whethermeta data is received and stored or not, the selection of meta data tobe transmitted to the composition/rendering module 4.

[0063] A number of rendering modi 19 are determined by meta datamanagement module 16 for the multimedia data elements and the meta dataelements. The rendering modi determine by which meta data elements amultimedia data element is influenced and which meta data elements haveno effect on it.

[0064] The rendering modi 19 is explained by the following example. If auser selects via the browser 2 the transmission of a video sequence fromserver A1 and a 3D object from server A2, and if the server A1associates a light source to the video sequence and the server A2associates a reflectance map to the 3D object, knowing that said lightsource and said reflectance map are transmitted respectively togetherwith said video sequence and said 3D object to the device 1, therendering mode may be to let the light source illuminate only the 3Dobject and not the video sequence and, furthermore, to prevent fromreflections from the 3D object onto the video sequence.

[0065] Furthermore, the rendering modi 19 may not only determine whetherthere is an influence between meta data and multimedia data, but mayalso determine how a meta data element influences a multimedia dataelement during composition/rendering. Therefore, the rendering modi 19may define the spatial resolution of meta data used, the type andnumerical precision of used light distribution or colour models, or anyother parameters defining the influence of a meta data to a multimediadata element.

[0066] Composition/rendering module 4 positions the multimedia data in acommon support space assisted by the user thanks to the browser 2. Thecommon support space can be a 2D plane, a 3D space or any other commonspace. The visual parts of the multimedia data are rendered consideringthe photometric meta data and the rendering modi 19 determined by metadata management module 16. Audio data are composed of at least one audiosignal.

[0067] We are going now to describe through a non-limitative embodimentthe treatment of the request of a user.

[0068] When a user wants to display an object on display 6 whilewatching a video movie for example, he selects the requested objectthanks to the browser 2. The requested object is issued from one of theremote server A1, A2, . . . , Aj, . . . , An. The request is sent frombrowser 2 to the interface module 5 and through the transmission linksC1, C2, . . . , Cj, . . . , Cn. Depending on the type of networks it isa broadcast request or a unicast request and the interface moduledepends also on the type of network. The interface module can alsoinclude several sub-modules if the links C1, C2, . . . , Cj, . . . , Cnare of different types as explained earlier.

[0069] Then the request is received by the transmission module 8 of oneof the servers A1, A2, . . . , Aj, . . . , An. The requested data isthen extracted from the database with its associated meta data. Thesemeta data have been generated by the meta data management module 18 oris stored in the data base 17. Then the requested data and theirassociated meta data are respectively sent to the multimedia coder 9 andto the meta data coder 10. Once coded, both are associated together inthe transmission module 8. The type of association depends on theprotocol used on the links C1, C2, . . . , Cj, Cn. They can beassociated for instance according to TCP/IP protocol if the links areinternet links.

[0070] The coded data are transmitted to the device 1 through thetransmission links Cj and received on the transmission module 14 ofdevice 1. The transmission module demultiplexes the received data andmeta data according to the protocol used on the links. Then the data andthe associated meta data are respectively decoded by the multimediadecoder 12 and the meta data decoder 13. The data are then sent to thememory 3 and treated by the meta data management module 16. This metadata management module 16 determines the rendering modi to be used bythe composition rendering module.

[0071] Then the composition and rendering module 4 treats the data andthe modi and sends the composed data to the display 6.

1. Device (1) to generate mutual photometric effects between a pluralityof visual data sets, comprising means to position the visual data setsin a common support space characterized in that it comprises: means (5,11, 12, 13, 14) for receiving and demultiplexing said visual data setsand photometric parameters respectively associated with said data sets,means (16) for defining said mutual photometric effects to be generatedfor these photometric parameters, composition and rendering means (4)for positioning said visual data sets in said common support space andapplying said effects defined for said photometric parameters from atleast one of said visual data sets to at least one other of said visualdata sets so that at least one visual data set influences one othervisual data set in said common support space.
 2. Device (1) according toclaim 1 characterized in that the means (16) for defining the effects ofsaid photometric parameters manage their storage, their numericalprecision, their spatial resolution and/or their rendering together withvisual data sets.
 3. Device (1) according to any of claims 1 or 2characterized in that the photometric parameters are illumination andreflection properties, the illumination properties being preferablyselected among the spatial intensity distribution of light, the positionof light sources, the shape and the color of light sources and thereflection properties being preferably selected among the surface color,the specular color, the reflection geometry and the polarization. 4.Device (1) according to any of claims 1 to 3 characterized in that thecomposition and rendering means for applying the effects described bysaid photometric parameters generate special effects like preferablyshading, cast shadows, self shadowing, specular reflections,inter-reflections or color effects.
 5. Device (1) according to any ofclaim 1 to 4 characterized in that the visual data sets are still data,dynamic data, compressed data, raw data, panoramic data,three-dimensional data and/or image data.
 6. Server (Aj) for deliveringmultimedia data including visual data sets associated with respectivephotometric parameters for said visual data sets, characterized in thatit comprises means (18) for managing the storage and conditioning ofphotometric parameters for said visual data sets, means (8) fortransmitting over a transmission link (Cj) said multimedia data bundledwith their associated photometric parameters, said multimedia dataincluding means to display at least one of the visual data sets withanother of said visual data sets in a common support space.
 7. Server(Aj) according to claim 6 characterized in that the means (18) formanaging the storage and conditioning of photometric parameters definesthe spatial resolution, the numerical precision, the coding mode and thekind of association to multimedia data of the photometric parametersconsidering type of service, type and state of transmission link (Cj),type and state of the device (1) connected to the server (Aj) via thetransmission link (Cj) and/or profile of user using said connecteddevice (1).
 8. Multimedia system characterized in that it comprises atleast one server (Aj) according to any of claim 6 to 7 for deliveringmultimedia data including at least two visual data sets associated withphotometric parameters for visual data sets, means (Cj) for transmittingsaid visual data sets and their respective photometric parametersbundled together from said server to at least one device according toany of claims 1 to 5, a device (1) according to any of claims 1 to 5 togenerate mutual photometric effects for applying the effects of saiddelivered photometric parameters of at least one of said deliveredvisual data sets to at least another of said delivered visual data setsso that said visual data sets influences said other visual data set in acommon support space.
 9. Multimedia system according to claim 8characterized in that the means (Cj) for transmitting said visual datasets and their respective photometric parameters bundled together arebroadcast means, unicast means, multicast means, connection-orientedmeans, connection-less means is and/or media distribution means. 10.Method to generate mutual photometric effects between a plurality ofvisual data sets, wherein one positions the visual data sets in a commonsupport space, characterized in that: one delivers multimedia dataincluding at least two visual data sets respectively associated withphotometric parameters for visual data sets, one transmits, bundledtogether, said visual data sets and their respective photometricparameters to at least one managing means, one defines dynamically saidmutual photometric effects for the photometric parameters and theirassociated visual data sets, and one applies said effects defined bysaid photometric parameters from at least one of said visual data setsto at least another of said visual data sets so that said visual dataset influences said other visual data set.